fix(agents-api): Fix for temporal running out of history size #687
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #678
Closes #679
Signed-off-by: Diwank Singh Tomer [email protected]
Important
This PR introduces remote data handling in the agents API to prevent Temporal from running out of history size by storing large inputs and outputs in a blob store.
save_inputs_remote_fn
andload_inputs_remote_fn
insync_items_remote.py
to handle large inputs by storing them in a blob store.save_inputs_remote
andload_inputs_remote
as activities.RemoteObject
,BaseRemoteModel
, andRemoteList
inremote.py
to manage remote data storage and retrieval.StepContext
intasks.py
to useRemoteObject
forexecution_input
andinputs
.store_in_blob_store_if_large
andload_from_blob_store_if_remote
instorage_handler.py
to manage large data.auto_blob_store_workflow
decorator for workflows to handle remote data.TaskExecutionWorkflow
intask_execution/__init__.py
to use remote activities for input/output handling.task_execution/helpers.py
to support remote data handling.blob_store_cutoff_kb
inenv.py
anddocker-compose.yml
to 64KB and 128KB respectively for better data management.This description was created by for f7879d3. It will automatically update as commits are pushed.